QTM 385 - Experimental Methods

Lecture 03 - Natural and Quasi Experiments

Danilo Freire

Emory University

Hello, everyone! 👋
Hope you’re all doing well! 😉

Brief recap

Last time, we saw that…

  • A good research question should produce knowledge that solves real-world problems and guides policy decisions, with a practical and credible research design

  • Theory is essential in research design, whether implicit or explicit, as it helps generate hypotheses, informs design choices, and guides inference strategies

  • Operationalisation is the process of translating theoretical concepts into measurable variables, such as turning “social isolation” into the frequency of social interactions

  • Pre-registration involves filing research designs and hypotheses publicly to reduce bias, improve credibility, and distinguish between pre-planned and exploratory analyses

  • The reproducibility crisis in science highlights the need for transparent and replicable research, with pre-registration and pre-analysis plans helping address this issue

  • The EGAP research design form provides a blueprint for creating robust research designs, covering key components like hypotheses, population, intervention, outcomes, and analysis

  • The MIDA framework (Model, Inquiry, Data, Answer) helps researchers simulate and diagnose their designs before implementation, ensuring they can answer their research questions effectively

  • DeclareDesign is a tool that allows researchers to declare, diagnose, and design their studies using the MIDA framework, improving the quality and credibility of research

  • A two-arm trial is a common experimental design where units are randomly assigned to treatment or control groups, and the average treatment effect (ATE) is estimated

  • Alignment between research design and theoretical frameworks is necessary for generating credible and actionable results, even when experiments are not feasible

Today, we will discuss…

  • The experimental ideal: why we should keep it in mind even when we use observational data
  • Natural experiments: what they are and how they can help us estimate causal effects
  • Quasi-experiments: how they differ from natural experiments and the challenges they pose
  • The quasi-experimental toolkit:
    • Matching
    • Instrumental variables
    • Regression discontinuity
    • Difference-in-differences
  • Internal and external validity: the trade-offs between them
  • Cumuative knowledge: how different research designs contribute to it

The experimental ideal 🧑🏻‍🔬

A benchmark for causal inference

  • You already know by now that the most credible way to estimate causal effects is through randomised experiments
  • This is because experiments address confounding by randomly assigning units to treatment and control groups
    • By confounding, we mean the presence of unobserved variables (\(Z\)) that affect both the treatment and the outcome
  • This means that:
    • All covariates are balanced between treatment and control groups
    • The treatment is independent of potential outcomes (more on that in the next lecture)
  • Today, we will investigate this ideal a little further, and see how we can get closer to it even when we don’t have experimental data
  • But first, let’s see an example